Roadmap

Peregrine was created to optimally handle iterative and graph computation for which other map reduce frameworks such as Hadoop were non-optimal.

Specifically, the design of Peregrine is setup to fasciliate evential implementation of a Pregel style graph computation framework as well as FlumeJava style pipeline and optimizer framework. (Interestingly enough the author was already working on similar frameworks but via convergent evolution these ideas are better documented via these papers).

Further, a domain specific language similar to Pig and Sawzall is also being considered to democratize access to these types of resources. Making the runtime Java-only limits the potential audience and this framework can also be very useful for other types of developers including operations.

Contributors

Peregrine is developed at Spinn3r for use with our blog and social media crawler. We use it for machine learning and other IR tasks where performance is paramount.

The Peregrine Falcon

Peregrine was named after the Peregrine Falcon due to the author's love of the bird as well as his interest in science and nature.

The Peregrine is the fastest animal in the world: "The Peregrine is renowned for its speed, reaching over 325 km/h (202 mph) during its characteristic hunting stoop (high speed dive), making it the fastest member of the animal kingdom.

The Peregrine Falcon represents exactly what we're trying to accomplish with Peregrine. We want to be the fastest MR implementation but at the same time remain small, agile, elegant, and beautiful.